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/ABSTRACT 

The process of writing test items is ainalyzed, and a 
blueprint is presented for an authoring system for test item writing 
to reduce invalidity and to structure the process of item writing. 
The developmental methodology is introduced, and the first steps in 
the process are reported. A historical review traces the advcinces 
made in the field and the formal systems developed for reviewing test 
items. A Computer Aided Item Construction Project has been initiated 
to integrate the results of test writing research into cin Authoring 
System for Item Construction (ASIC) program. An existing systems 
development method was used to guide the process of developing the 
authoring system. This method. Information Systems work and Analysis 
of Changes (ISAC), starts with analyzing the needs, ideas, and 
problems of those who handle information. Only this first stage. 
Change Analysis, is described in this report, steps in change 
analysis are: (l) problem analysis; (2) description of* the existing 
situation; (3) making a test plan; (4) choosing an item format; (5) 
creating the test item; (6) checking the items; (7) study of change 
alternatives; (8) developing a blueprint; (9) choice of the item 
format; (10) generating items; and (11) checking items. Problems of 
item writing are described and listed in two tables with reference to 
whether an authoring system would contribute to their solution and 
the validity of the test. In all, 4 tables and 12 figures illustrate 
aspects of item writing. (SLD) 
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Abstract 

The process of writing test items has many sources of 
possible invalidity. In this report, the process of item 
writing as it is used today is analyzed, and some sources of 
invalidity are described. In order to reduce the invalidity 
and to structure the process of item writing an authoring 
system for test item construction is proposed. 
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Towards an Authoring System for 
Test Item Construction 

Improvement of the quality of education is a permanent topic. 
One way to improve qualiicy of education is to improve 
testing. This report deals with the very first stage of 
testing, namely the construction of test items. 

The first part of this report gives a historical review 
of new ideas about test item construction. The second part of 
this report presents a blueprint for an authoring system for 
item writing. A developemental methodology (ISAC) 
introduced and first results are reported. 

Historical Review 

Improving the quality of test items has always been of 
great concern to test developers and researchers using tests 
to collect data. With the emergence of the criterion- 
referenced testing movement the concern about the quality of 
test items has become a focus of special interest . The paper 
of Glaser (1963) marks the beginning of the criterion- 
referenced testing movement . The interest in criterion- 
referenced testing coincides with changing demands upon 
testing . These changes were concerned with the search for 
assessment methods that could provide information for 
individual and programmatic decisions aris ng from specific 
competencies . Hambleton & Rogers ( 1986 . p. 208 ) put it like 
this: "Norm— referenced tests were judged to be inappropriate 
because they provided information that facilitated 



Item Construction 
4 



comparisons among examinees on broad traits or constructs. 
Norm-referenced tests were not intended to measure specific 
competencies . And even if items in a norm-referenced test 
could be matched to competencies, typically there would be 
too few test items per competency to permit valid criterion- 
referenced test score interpretations." 

The growing interest in criterion-referenced testing as 
a basis for student selection, policy making and research has 
changed the demands on requirements of the test item. In the 
criterion— referenced testing approach the test consists of a 
representative sample of test items from the universe of test 
items describing the domain of knowledge or skills to be 
tested. Therefore, there should exist a direct relationship 
between the item and the domain. Moreover ,^ it should be clear 
what cognitive ability is measured by the items in the test. 
The demands made upon the test items by the criterion- 
referenced testing approach emphasizes shortcomings of the 
traditional item writing methods. In the latter a direct 
relationship between domain and item cc^tent is not always 
present, and it seems to be very difficult to classify items 
according to the cognitive ability they measure (Baker, 1974: 
Zwarts, 1982). 

Outside the criterion-referenced testing movement other 
prominent researchers also expressed their concern about the 
quality of test items . Problems with respect to the quality 
of test items have been formulated by Bormuth ( 1970 , p. 6) : 
"At the present time we seem to be in the position of having 
to accept the assertion that a test measures whatever the 
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test writers claim it measures without recause to definitive 
independent evidence." We find that the problem boils down to 
the fact that there is no existing methodology of test item 
writing. Writing test items still is considered a skill to be 
learned by experience. According to Roid. Haladyna. and Finn 
(1980) methods for writing test items, particularly for 
criterion— referenced testing, are needed that are (1) based 
on a logically defined relationship between the instructional 
materials and the test items and (2) capable of producing 
items that can easily be replicated by many test developers. 

Guttman and Schlesinger ( 1967) proposed the mapping- 
sentence method foi specifying an item domain. The mapping 
sentence method is based on facet design theory. A mapping 
sentence is produced by content analysis; it pays attention 
to the relationship between the required achievement and the 
cognitive abilities needed to perform adequately. An example 
of a mapping sentence is provided by Roid and Haladyna (1982. 
p. 139) and is shown in Table 1. 



Insert Table 1 about here 



This mapping sentence is derived from an instructional 
objective and can be used as a source for a large numbe:. of 
items . The objective is to compute and distinguish 
correlations between two variables (see Table 1). The mapping 
sentence consists of fixed and variable sentence parts . A 
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variable sentence part is called a facet. Items are 
contracted by filling in the facets provided by content 
analysis. This means that all items have the same syntactical 
structure. In this example a multiple choice item is 
produced, including the foils that are directly related to 
the results of the content analysis. This means that the test 
responses, including the foils can be used to measure 
achievement. Millman (1980) is convinced that' the development 
of mapping ser^tences and the item generation from these 
sentences can be structured. 

Since a domain is described by a set of mapping 
sentences and each mapping sentence consists of sets of 
facets, the description of a domain can easily result in 
large amounts of documentation, and the use of a computer to 
store the mapping sentences and the sets of facets would be 
inevitable. The generation of the items by computer is then a 
logical next step. 

Hively. Maxwell, Rabehl , Sension, and Lundm (1973) 
used a so-called item-form technique to generate items 
directly related to objectives and content. Defining the 
domain with this technique has some resemblance with the 
mapping sentence method. Like a mapping sentence, an item 
form is a genuine operationalisation of an objective. Hively 
et al. (1973) developed a format to display item forms. An 
example of an item form is shown in Figure 1. 
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Insert Figure 1 about here 



The item form consists of an item form shell (to be 
compared with fixed sentence parts of a mapping sentence). 
Figure 1 shows an item form for the concept of comparing 
cardinality of sets by one-to-one correspondence. The item 
form provides very detailed information about how the items 
must be constructed, but at the same time provides a source 
for numorous different items. Again it is possible to store 
item forms (written as a computer program) in a computer 
(Roid and Haladyna. 1982) leaving the actual item generation 
to the machine . 

The linguistic approach as introduced by Bormuth (1970) 
is another attempt to structure the item construction 
process. This technique constructs questions by selecting 
sentences from instructional text. The selected sentences are 
transformed into items. The selection of sentences is based 
on the presence of so-called high-information words. High- 
information words are words that are relatively rare in 
American English and have a low text frequency. When 
sentences are selected, their semantic structure is used to 
transform them into items An example of this procedure is 
presented in Figure 2 . 
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Insert Figure 2 about here 



Roid et al (1980) developed an algorithm for generating 
test iters by means of the linguistic technique. 

Advances are made in preparing handbooks to aid novice 
(and experienced) item writers and reviewers , and formal 
systems have already been developed for the rev^^ewing of test 
items (e.g. Herman & Winters. 1985). The use of these 
handbooks contributes to the clarification of the item 
writing process. 

In reviewing the emerging technologies of item writing. 
Roid and Haladyna (1980) pointed out, that it woul be best 
to consider a continuum of methods from (a) the informal, via 
(b) the objective-based to (c) vhe algorithmic method. 
Depending on the test requirements as well as time and 
resources available, the item writer should cnoose the most 
appropriate method. 

But the application of all these different techniques 
requires a lot of organizational talent. It also takes an 
enormous effort to reach an adequate level of efficiency and 
skill in using these (new) methods. For these reasons, the 
author started a Computer Aided Item Construction (CAIC) 
project to integrate alle these techniques and methods into 
an Authoring System for Item Construction (ASIC) program. 
Such an authoring system will contain procedures to provide 
all information needed to write items. This includes, e.g., 
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information printed in handbooks as well as information on 
constructing item forms. The information will be stored in a 
data base, and can be consulted by an item writer in every 
processing fase. In addition. the system will provide 
decision rules for every step the item writtr tckes. In this 
way. the system assists in declaring mapping sentences, or in 
defining the replacement sets for an item form. 

Two basic questions for the CAIC project are: 

1. How to describe the process of item construction 
as detailed as possible. 

2. What are the implications of automating every 
subprocess . 

The purpose of this report is to answer the first 
question. If we consider item construction as an integrated 
process of information handling, we best use an information 
systems development methodology to describe this process. 
Using an information systems developement methodology also 
enables us to answer the second question, because the 
methodology will provide detailed describtions of every 
subprocess . 

Item construction in an authoring systems environment 
will enable every item writer to use expert knowledge on item 
construction techniques. This means that an item writer can 
really use the whole continuum from informal to algorithmic 
methods . 

Using an authoring system creates the possibility to 
link item construction and computer-aided testing. In the 
future, this may result in a revolutionary change in the 
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appearance of items. Traditional items can be replaced by 
computer simulations or even video games (Jones. 1984; van 
der Linden. 1985). This seems a promising future for ability 
testing, especially for testing higher level abilities. 

Developing an Authoring System. 

An existing systems development method will be used to 
guide this process. This method is knovm as the ISAC-method 
(Lundeberg. Goldkuhl & Nilsson. 1980). ISAC means Information 
Systems worV and Analysis of Changes. The method starts with 
analyzing the needs, ideas and problems of those who handle 
information. The method ends with the specification of tasks 
chat must be conducted by hand and computer programs. The 
method consists of a number of feasible arid cohesive steps. 
Going through every step, a documentation file (describing 
the system as it is developed) is build. Finally, a fully 
described system can be developed, and every detail is 
documented. The reasons why this method is used are: (1) one 
does not have to be an experienced systems designer.. (2) it 
permits adjustments of the procedures to the subject under 
study, (3) the method is orientated towards the activities 
that need support, and (4) the entire process of systems 
development is cut into manageable steps. 

The method provides a step-wise development of an 
information system. During these steps, documentation is 
created that will function as the backbone of the system. The 
documentation consists of graphs or charts reflecting the 
information flows. The graphs indicate who is responsible for 
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the generation of information. In this way the system is 
described in every little detail. The graphs are accompanied 
by text pages that explane the short descriptions used in the 
graphs (see e.g. Figure 3). Outside the box on the left hand 
side in Figure 3 we can see the input at the top and the 
output at the bottom. The parallelograms contain a short 
description of the kind of information needed for an 
activity, or the product of an activity. The parallelograms 
with a double upper line contain information provided by e.g. 
the systems database and materials supplied by the user. The 
parallelograms with a single upper line just contain 
information not provided by the user. The connections between 
the different activities are represented by lines. Thin lines 
indicate transport of non-user provided information: bold 
lines indicate a mixture of user generated and system 
generated information. Activities are represented by the 
small circles. 



Insert Figure 3 about here 



The first stage in the ISAC-method is change analysis . 
During this stage it is assessed whether the development of 
an information system is really the best solution to a 
particular problem. The following stages in the method are 
developmental stages (activity study: information-analysis; 
data system development: adjustment of means). This report is 
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only concerned with the change analysis . Later reports will 
cover the remaining stages of ISAC, 

Change Analysis 

Before an information system is developed a change 
analysis is performed. Change analysis attempts to analyze 
what kind of changes (in fact, improvements) of activities 
can be made in real-life situations. The reason for 
conducting a change analysis is that one is not only 
interested in tackling the symptoms which are caused by some 
problems, but also in finding and analyzing the problems 
themselves . 

The results of change analysis can directly be i^'-ed in 
the next steps of systems development, and. in fact, can be 
used as tools to speed up the development. If change analysis 
leads to the conclusion that the needs and problems of the 
users will not be solved by the development of an information 
system, then it would be better to change the strategy. 

Change analysis consists of three steps. First, a list 
of problems is formulated. Next, the existing situation is 
described, and finally, a description of the resulting 
information system is given. 

Problem Analvsis 

Starting a change analysis m'^ans starting a detailed 
description of the existing problems. These problems can be 
listed in a Problem i^ist . 

ER?C 1G 
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The problems listed in Table 2 can be grouped into four 
categories which are given in Table 3. 



in-?ert Table 2 and 3 about here 



Descrip t ion of the Existing Situation 

The description of the existing situation can be 
conducted systematically by means of Activity Graphs (A- 
Graphs; and Text Pages according to the ISAC-inethod 
(Lundeterg,, et al, 1980). Figures 3 to 12 show Activity 
Graphs or. the left half of the figure and the Text Pages on 
the right half. The Activity Graphs are used to 
systematically document the activities involved in the item 
writing process (see Figure 3). This process cou. d give the 
impression that item writing is, in fact, a systematically 
structured procedure. The contrary often is the case. Only 
vague guidelines exist which leave large blanks to be filled 
up by the item writers experience (and subjectivity). 
Therefore, the existing situation (shown in Figure 3) can 
only be described by an overview of all recommendations 
agreed upon by leading experts (e.g., Thorndike, 1971; Ebel, 
1972; Nitkc 1983). 
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Making a Test Plan . 
In todays iteoa-writing practice, writing itself is 
considered to be part of the test development. This means 
that the need for a concrete test is the starting point. This 
differs from computerized testing where the generation of an 
itembank can be seen as a separave activity (van der Linden. 
1986). Information on (1) the content area of the test. (2) 
the cognitive objectives, and (3) the test requirements are 
gathered in a * ^stplan (see Figure 4). 



Insert Figure 4 about here 



The formation of a testplan is supported by guidelines. 
These guidelines lead to a two-way table where an outline of 
the domain is placed on one axis of the table and the 
cognitive objectives along the other (Mehrens & Lehmann. 
1973). In this way it is possible to classify item types in 
the cells of the table simultaneously by domain and 
objective. The next thing to do is deciding on how many itemr 
have to be written for every item type in the table. 

This procedure seems to be a valid one, since only the 
number of items and the classes have to be determined. 
Problems arise., however, when the outline of the content area 
and the detailed description of the objectives have to be 
obtained. It is well known that it is not easy to extract 
information from educational objectives, course descriptions., 
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information from educational objectives, course descriptions, 
id. No rules for this kind of activity are available. This 
means that every individual item writer has to rely on 
his/her own methods and experience. This, in turn, means that 
different item writers may have different results at the end 
of the procedure and thus come up with different plans for 
the same test. There are at least two possible kinds of test 
plans. One is a comprehensive review of the curriculum, which 
includes all the topics and behaviours connected to the 
course. The other is a selection of the materials based on 
the most important topics in terms of learning possibilities 
and acquisition of desirable behaviours. Since the latter 
kind of test plan may cause large discrepanties between item 
writers, we prefer the 'comprehensive' type of test plan, a 
choice also in line with the principles of criterion- 
referenced testing. The problems grouped in G2 and G3 (and 
partly in Gl) arise (see Table 2 and 3) at this stage. 

Choosing an Item Format . 

The next decision to be made is the one on the format 
of the items in the test. This decision can be made after 
consulting the test plan and the information on different 
item formats, as is shown in Figure 5. 



Insert Figure 5 about here 
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This choice is also a possible source of disagreement 
among item writers. See the problems mentioned in G2 (of 
Table 2 and 3) . 

Creating the Test Item . 

Up till now the actual creation of the test item was 
seen as the exclusive domain of the test writer. The ideas 
for the item text had to come out of his/her creative mind, 
and there were no rules available to guide this process. The 
writing process (see Figure 6) can be based on (a) the 
information from the testplan. (b) the information from the 
chosen item format ,^ (c) the information on how to write items 
(e.g..^ old items),, and (d) the information from items created 
in another setting. 



Insert Figure 6 about here 



After the item text is created., graphics can be added 
if necessary. Before the item is filed, the lay-out must be 
checked and instructions can be added. 

The freedom of the item writer in this phase can result 
in a serious bias comparable with the problems clustered -"'n 
G2 and G4 (see Table 2 and 3). To improve the construct 
validity of test item^ , strict rules for this stage of the 
writing process are needed. 

ERIC 20 
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Checking the Items » 

Before the items can be used in a test, a final check 
has to be made. Figure 7 shows what activities are needed for 
this check. First of all. the fulfilment of the requirements 
collected in the testplan has to be checked, and possible 
deviations from the testplan can be corrected . Next . the 
items have to be checked with the help of available 
checklists, and finally a check on spelling and grammatical 
errors is made. If the lay-out is considered to be correct, 
then the item can be stored. Any rejected item is placed in a 
pool and can be used again as input for a following writing 
stage . 



Insert Figure 7 about here 



In this stage a number of problems may be encountered. 
The proposed checking procedure is often not precise enough 
to detect every error in the items. In fact, all the problems 
mentioned in G2; G3 and G4 (see Table 2 and 3) should be 
taken care of in this stage, but usually this is not done. 

Study of Change Alternatives 

The last stage of change analysis consists of a description 
of the future system. The description enables us to compare 
the desired situation ' with the existing situation. The 
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descript:?on of the system starts with a listing of a number 
of goals of the system (see Table 4); Activity Graphs and 
Text Pages are used for this description. 



Insert Table 4 about here 



A great advantage of an authoring system for item 
writing is that a structure can be imposed on the procedure. 
Every step (oS shown in Figure 8) can be guided by rules, and 
some parts of the process can even be handled by applying 
heuristics and algorithms. 



Insert Figure 8 about here 



In principle the steps for developing test items do not 
differ much from the steps described earlier. Only the 
activities in every step are different. 

Developing a Blueprint . 

The development of a blueprint does not differ to much 
from the development of a test plan (compare Figure \ with 
Figure 9). The differences are mainly a consequence of the 
fact that the items are no longer written for one specific 
test. This means that test requirements are no longer part of 
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the constraints for the blueprint. These requirements can be 
fullfilled during the testconstruction. 



Insert Figure 9 about here 



The development of the blueprint will lead to a two way- 
table with conten'. information as the row headings and 
cognitive objectives as column headings . The construction of 
this table is guided by rules (heuristics and/or algorithms), 
that must ensure the replicability of the table. This means 
that the freedom of the item writer must be limited in favor 
of the replicability of the results. An addition to the 
information in the table is the information on the difficulty 
level cf the text material used in the course (also* called 
readability level). The difficulty level is an index that can 
help the item writer to write items matching the index. 
Furthermore, the index is valuable when deciding what 
cognitive abilities should be tested. 

The cells of the table will be filled with information 
describing the items to construct as detailed as possible. 

Choice o f the Item Format . 

In Figure 10 we see that for every cell of the table of 
specification an item format must be chosen. This choice must 
depend on tha information in the cell. The test requirements 
for one specific test are no longer part of the decision. 
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Instead, a whole system of decision rules is added to aid the 
decision on the item format. The choice of the item format 
for a cell of the table of specification thus will be a 
compromise between the choice according to the item 
specifications in the cell, and the choice as a result of the 
use of decision rules. If the ultimate dacision is left to 
the item writer, it is now clear on what grounds it was made. 



Insert Figure 10 about here 



Genera t i ng Items . 

The table of specification and the item format 
connected with every cell in the table., are the input for the 
decision on what strategy should be applied to generate items 
(see Figure 11), and for each chosen strategy there will be a 
set of rules and additional information to structure the 
process . 



Insert Figure 11 about here 



The freedom of the item writer is restricted in favor 
of clarity and control of the process, and the severity of 
these restrictions will differ foi each strategy. The 
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strategies themselves vary along the continuum proposed by 
Roid & Haladyna (1980). For every strategy, detailed rules 
should be worked out. 

A very practical addition is the generation of graphics 
that could be integrated in the item writing process. This 
will save time, and results are immediately available. In the 
long run. graphics could become a very imported part of the 
system. We predict a wide range of new possibilities in item 
generation processes when in computer aided testing graphics 
are combined with interactive video. 

Checking Items . 

The checking of items as shown in Figure 12 can be 
supported by such information as checklists: spell checkers: 
and. in the future. word processors using artificial 
intelligence to check on grammar: id. 



Insert Figure 12 about here 



Conclusions 

To decide whether the problem? described in Table 2 
(and grouped in Table 3) can be solved best by developing an 
describ-^d authoring system, we must also consider alternative 
solutions. 
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Gl It;em Construction Is Tlmft Consuming 

Time can be reduced by integrating text and graphics 
generation (see Figure 11). When a word processor and other 
computer programs are used to check the items, the time 
needed to develop an item will be shortened. The time needed 
to search for information on various subjects (e.g. item 
formats or generation rules for multiple-choice items) will 
also be reduced by storing all information in a database. 

It will cost time to prepare a test plan, as it will 
cost tine to prepare item generation according to a chosen 
strategy (see Figure 11). But once time has been invested, it 
will be relatively easy to develop more items of the same 
quality. And once an item set has been developed, it can be 
used many times. The question whether this automatically 
implies tha& item generation should be automated cannot be 
answered yet. The suggested item writing techniques (Roid & 
Haladyna. 1980) can all be used without a computer. There is 
still work needed to show that the use of a computer is 
possible and will speed up the process. 

The conclusion for proble-^ group one therefore is that 
there can be a reduction in time needed to generate items 

G2 Ite.Tiwriter Bias Reduces t^e ValidjLty of Items 

This group of problems has been treated very thoroughly 
in the developed system. The process will be guided by rules 
to guard against itt^m writer bias. The construction of the 
table of specification and the item generation itself are 
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guided by heuristics and algorithms to reduce differences 
among item writers, and differences in experience will be 
reduced by the information available during the whole 
process . 

For some evidence that the item writing methods reduce 
item writer bias see Roid and Finn (1978; 1980). 

G3 Content and Construct Validity Often Questionable 

This problem is solved by guiding the building of the 
table of specification (see Figure 9). By making a 
comprehensive blueprint and checking the difficulty level of 
the instructional materials we are quite certain that the 
overall content validity of the items improves. 

The construct validity of the items is a somewhat 
different matter. The relationship between students 
performances on an item and the ability the item is supposed 
to measure is not easy to establish. Up till now, the best 
help available is a set of taxonomies to classify items 
according to the ability they measure. The problem is that 
these taxonomies can only be used with existing items This 
means that if we want to use them while generating items, we 
have to compromise We could, for instance, use classified 
sample items to start from. Having gent^ated some items, we 
could then check the ability they measure by classifying them 
again. 
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G4 Items Often Show Editorial Mistakes 

Checking items using computer prog-* nay reduce the 
number of mistakes and save time. With the use of artificial 
intelligence in combination with a word processor, it should 
be possible to eliminate most editorial mistakes. 

G5 Item Writing as an Act of Creation 

Subjecting the item writing process to rules, it would 
appear that there is no more room left for the creative item 
writer. This is not true. The rules allow control of the 
generation of items. But there still is a need for creativity 
when the item itself has to be made. Therefore, it would be 
better to conclude that there still is a lot of creativity 
involved in generating items, but that the generation process 
has a more scientific basis if the results can be replicated. 
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Table 1 . Example of a mapping sentence developed from an 
instructional objective (Roid & Haladyna. 1982). 



<}hjrHi%rS 

1 Given 4 NCI uf ofikfcd pairs of valuer on vanabk X and Y <.orre(.tly cakulitc r^, 

2 Given siaicmcniJ abcui ihc PcjrMin poini bisenal. P'". and rank order corrclj 
itofi coefficicms. the Jeamer will identify iho^e ihat accuralcly compare jnd <.i»nU4s« ihc varioux 
measures 

Mup/itiiK Stnttnce 

/A Presentation 



Given J < 



ftinn 
table 



2 prtisc pjN^j^e 
A vet ol ordered pairs of values on 
vanablev 



Content fonii 

1 verbatim 

2 concept 



I ^ pjirs 
\ 4 pjirs 
I 5 pairs 
Vanable X 

1 one di^ii 

2 iwo digit 
three digits 



Vjnabic y 

1 4ine di^it 

2 two dibits 

^ three digits 



the simJent will select the correct value of Iroin j %<t of ^ftemjii^es ih.u v if> niih respeii lo 
fO MultiphCdiion of ^ 



h Type ol store i 

1 devtation V 

2 rjw \ 



signed numbers 

1 t.orfe(.t 

2 iniorreLi 



H Division of SP (A >) i 

1 no / 

2 ves thjt IS SP ( \ M n\ 
I Square timm of SSI \ ) SSI) \j 

1 vcs > 

2 no \ 
J Tvpe ol udit 

1 no unii 

2 lincji uiiii 
^ squdte unit j 
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Table 2. Problem List 



PI item construction is time consuming 

P2 quality of items differs because of itemwriter bias 

P3 expertise and experience of itemwriters differs 

substantially 
P4 items are not always linked with curriculum, 

P5 items are not always linked with objectives 

P6 items don't frequently satisfy layout rules 

P7 items don't frequently satisfy construction rules 

P8 items often contain grammatical mistakes 

P9 items often contain spelling mistakes 

PIO items often contain deficiencies of style 
Pll items containing graphics cannot be constructed in 

one construction step 
P12 itemwriting is considered too much an act of creation 

and lacks a systematic approach 
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Table 


3, Problems grouped in clusters. 


Gl 


PI :P11 


G2 


P2;P3 


G3 


P4;P5 


G4 


P6:P7:P8:P9:P10; 


G5 


P12 



ERJC 



Item Coiiiitruction 
31 

Table 4, Goals of the authoring system. 



Gl the content of the items must have a direct and 

controlable relationship to the learning material 
G2 doubts about the cognitive ability an item measures 

should be minimized. 
G3 it should be possible that different item writers 

produce items not differing in quality 
G4 the item writer should have on-the-spot access to 

guidelines for item writing 
G5 the itv>m writer should have on-the-spot access to 

means and tools to check the quality of items 
G6 integration of graphics and the item text should be 

made very easy. 

G7 the system should have an open structure so that new 

modules can be attached at any time 
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XTtM rC*N aULL 



ttlM rcM t It 
Corr«ipond«ftc« 
euwAi oifcitxrTioii 

Th* child givtA Aitb*r two or t\r** ••ti oi "sauQt^rt 
••eh h«vlD9 •pproilMtolr 20 m—b9tt (or l*ti> T&* lata 
Mf h«Tt tlt^ &u)iM7 of t)«nb«r« or th*r 9»T dlfftr 

o&« Mabcr Tho child l* ••kMl to ihow wh*th*r or not tht 
••t« h^ro th^ •••• &uffl)«r of »*ftb«ri without count irg 

•TUWLV* JUTD ltC«»0«»ft CNAMCTmKttTlCf 
Constant for •U CtUi 

Only tt^Ad^rd *couat*r** (tn^Jl colored dlatca) tr* ua«l 
t«cb att ct coiuittri 1* • dlif«r*it color (r*d gfn or 
yollotr) 

t>l«tiogulthia9 b«tw««a ctllt 

Xuab«r of tttt ceQp«r*d C?¥o or thr»#> Wh#tfc»r or not 
tho ittt h«r« th« tuf atiAUr of ob}*ct« llooroiisat* nu>- 
b«r of objtcta ia oAcb ••t {•tocut S *ix>ut U. or tDout 2:) 

Vkrrin? with C*ll« 

No y«rl«tloa 

CSU. kATKXX 



IgprotlMtt 
.Tu«t>«r of 




of $«tl 
»AJ*d 

SoUtloao 
»»• - 




Equ«iitr 

'c 


RtUtlon 

9b*' ^c 


s 


<n 


(4) 


(7) 


CO) 


()3> 


13 


(3) 


(S) 


(1) 


(11> 


CM) 


31 


(3) 


(4) 


(t> 


(12> 


(IS) 



HAmXAU 

1 Stt of cou'itvrc (•) 

2 Stt of counttri (b) 

3 S*t of ccuntvri (c ) 



blUCTXONI TO S 

Pltct ch* Abova acta n*tr 
• Ithtr •dg* ^ar.d aiddl*) 

of th* toit board tt s^owr 
•bo?* Th*n aty 



Tf S bvglna to coji^ or 
*I don t knew hew. ' 

• •r 



Vhtn S h4t flaiah»d tty 

K**p • rusain? rtcord of 
k.. 5 (• — 



wh«t 5 dot* t&d ttfi 



S!iov »• If thtao (d) atta 
(point) ht>* tht ••s* ou>- 
of ••wMra 



M c]«aa you ptlr»d ob)*cta 
to tt)l If two avta htd tht 
i«0* nuA£>«r of a»«b«ri. 

?>•••• •')OV M if th«i* (d) 

aita tht au* noiDtr 

of o*ab«ri 

Do tA»f htv* t**.* lAAt 

n.tfi>«r of ••cbvra? 



S*t< of Counttra (•) • r*d (b) • 97»»». (c) • fallow 



Nuub«r 


of 


objacta ia 


aoch aot 






Ctll 


1 


(a) 


S 


(b) 


S 


(c} 




Ctll 


2 


(a) 


13 


(b) 


13 


(C) 




c*n 


3 


(•) 


2: 


(b) 


21 


(C) 




Call 


4 


(•) 


s 


(b> 


« 


<C) 




Call 


S 


(a) 


13 


(b) 


14 


(c) 




Call 


A 


(a) 


21 


(b) 


22 


(c) 




Coll 


7 


(•) 


s. 


(b) 


S 


(c) 


S 


Call 


« 


(a) 


13 


(b) 


13 


(c) 


13 


Can 


♦ 


(a) 


21. 


(b) 


21 


(c) 


21 


Call 


30 


(a) 


S 


(b) 


S 


(c) 




Call 


11 


(a) 


13. 


(b) 


IJ 


(C) 


14 


Call 


12 


(a) 


21 


(b) 


21 


<c) 


21 


Call 


13 


(a) 


S. 


(b) 


«. 


(c) 


7 


Call 


14 


(a) 


13. 


(b) 


14, 


(c) 


IS 


Call 


IS 


(•) 


21. 


(b) 


22. 


(c) 
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Script (d>- 

Callt 1 throuQh 4. 'two* 
Calla 7 throuQh IS 'th.'aa* 

Child ahojld at«to corractlr (y» or ao) whothar or •ot tho 
•••.a hava t:ia aua cuabor of oaab«r». Ea ahould alao carry 
ous a dacacc«bla or.o-t>^na pairing oparatloa 
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Figure 1., Example of an item form (Hively et al., 1973). 
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EXAMPLE OF ITEMS PRODUCED FROM TEXT 
Keyword Noun — Metamorphosis . 

a. Text Sentence(s): After hatching, all insects, ^ except the 

most primitive, go throug a series of 
steps in development. These steps are 
called metamorphosis . 

b. Items (Stem and Foils) Produced by Item Writers: 

(1) What are the series of steps in insect development 
called? 

(a) Maturation (c) Symbiosis 

(b) Metamorphosis (d) Meitosis 

(2) What are the steps insects go thiough in development 
called? 

(a) Metamorphosis (c) Larva 

(b) Arthropoda (d) Pupa 

(3) What arc a series of steps in development called? 

(a) Reproduction (c) Metamorphosis 

(b) Larvae (d) Changes 

(4) What are the serie:, of steps in insect develop.T.ent 
called? 

(a) Encrytid (c) Arthorpoda 

(b) Instar (d) MetaT^orphosis 

c. Foils Produced Algorithmical ly . • 

Growths 

Metamorpt^osls 

Types 

Activities 



Figure 2. Example of foils produced by the alqorithm of 
Roid et al. (1980) . 




:a 

lAl 

1A2 

2A 

3A 

3A1 

n2 

4A 

4A1 

4A2 

4A3 

4A4 



B — 

$A 

SAl 

SA2 

SA3 
6 

6A 
6A1 

6B 

6B1 

7 

7A 
8 

8A 

6A1 
eA2 



content aroa 
content outilne 
l««rnlng □•ttrlaii 
cognitl/« objoctlvts 
test rtqulrooentt 
number o£ ltea» m test 
tioQ av«ilabl« 
need for partial tests 
iniorcwtlon backup 
Information on creating e bluoorlnt 
on different ItwfJf^Js 



creating the test blueprint"" 



the teat blueprint 
U^S«"<Jo'°?,"" "^"'P'^" "»"<»i"^» th. cbolc. Of th. 

lnlor«tlon Iroa th. t..t blu.prlnt rea.rdln, tb. «rltl„, of It. 

cassis? ^^.T^^^^^ 

Xeodbsck on choaen Iteoionaet to te<t ki..*,. i . 
^bS^in'J^^foV.."""^^"' t.%t"£Le^i^I£^« 

provlsionel items 

checking the provlalonai iteraa 

iteos hftve to be adjusted or iffioroT^d 
Items have to be ad^ed to ffla'tKZllueprlnt 



itoffls ready for use 



Figure 3. 



A GhAPH and TEXTFAGE shewing an overview 
state of the art of item writing 



of the current 
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4A} 



BSU 
BS3 
BS2A 
BS3 



5r 



cont«ni tr«« 
t«st r*qulr«D«att 

Information on cr«*tlng • blutprlat 



conttnt •p«cAiic«tIoD" 

row headiest for two-w«y t»bl« 

vtlghlag th« loportanc* of ••ch oMactlv* in th« t.K»*.>x/^« i« 



th© t8»t blueprint " 



AllfffVUit 



Figure 4, A GRAPH anc VEXTPAGE shewing the construction of a blueprint 
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5A tent blueprint 
J* tsut r«quir«o«nti 
3A1 n;"5b«r of ittiu in teit 
3*2 tloo •v«il«bl« 
3A3 need tot parallel teiti 

iDforoeti-jn oo different itea fonaati 

dSauct teit deoandi — 

F61A specified teit deoandi 
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40 




SA the test blueprint 

6B chosen Itefflforcat 

4A3 information on iteowriting for a chosen item format 

6A rejected/f lav items and coonent 

8A1 items have to be adjusted or ioproved 

8A2 items have to be added to match the blueprint 



W71A 
W72 
W72A 
W73 



rr 



MOINB I)610UCT!C« 



/prwuioftiL' 

ltfit£ / 



writing the icem text 

item text and cootaents 

lay-cut of iteos and adding of graphics 

completed items 

add i -^tructioRs to iteos if necessary 



provisional items 



Figure 6. A G?A?H and TEXTPAGE showing the construction of items. 



ERLC 



41 



« n » 




inloraatJlon on checking the quality of itetDS 
provisional items 

catching the blueprint and the iteo set 
check i£ the iteo set matches tne Diueprint 

use checklist to ensure correctness of items with regard to chosen 
item foroat 

item set and cocnents on construction errors 
check the items on spelling and grammatical errors 
item set and cooraents from C82 and C63 
check the items on lay-out errors 

rcjectcd/fiaw items and coaoent 
items ready for use 



Figure 7. A GRAPH and TEXTPAGE showing the checking of the items. 
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lA content area 

lAl content outline 

1A2 learning oateri.xs 

2A cognitive objectives 

3A infcraation backup 

3AI infonsatlon on making a blueprint 

3A2 infonsation on different item formats 

3A3 information on itea writing foL a chosen fonoat 

3A4 information on checking the quality of iteos 

4A heuristics and algorithms 

4AI heuristics for the construction of a test blueprint 

<A2 heuristics for the creation of test items 

4A3 algorithms for the autoaated generation of test items 

4A4 heuristics and algoritjjns for the checking of iteias 

^ constructing the test blueprint 

SA table of specification 

SAl information from the table of specification regarding the choice 

of the item format 

5A2 Information form the table of specification regarding the 

construction of iteos 

5A3 matching the table of specification and the item set 

6 the choice of the item format 

6A consequences for the table of specifications from the choice of 

the item format 

6B item format to be used 

7 applying item writing techniques 
7A generated items 

6 checking the Items 

8A rejected items to be corrected 

Items 




u 

21 
3A 
4A 



F51A 

dSLB 

B52 

B52A 

B53 

B53A 

353B 

^ 



content ar«a 
cognitive objectives 
Information backup on blueprints 
heuristics and algorithms on blueprints 



content speciiication 

^^^llfc^^torii'eeSrJ-c'eir'' specification 
weight of a cell In percentages items 



table of spocilication 



/lAlf of 
xim / 



Figure 9. A GRAPH and TEXTPAGE showing the wanted construction o: 
the table of specification 
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3A Information backup on itea foioats 

4A heuristics and algorithas on item formats 

5A table of specification 

751 closing an item toroat tor every ceil of the tabid ot spacilication 

P61A item format for each cell 

P62 comparing the item format according to the cell based choice and 

the item format proposed by heuristics/algorithm 

P62A adjustments of the chosen item formats 

P63 deciding on the item format 

5X feedback to table o£ specitication 

6B item format for all ceUs of the table of specification 



Figure 10. A GRAPH and TEXTPAGE showing the procedure to chose 
the iten format 
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Information backup on writin? itoas 
heuristics and algorithas on writing items 
table of specification 

item format for all cells of tba table of specification 

clxosing a strategy lor writing the iteos for a cell oi th9 table 
of specification 

a item writing strategy for each cell of the table of 
specification 

generating items by applying a technique belonging to a specific 
strategy 

generating plain item texst 
adding graphics to item text 

generated items 



ERIC 



Figure 11. A GRAPH and TEXTPAGS showing the construction of items in 
the- new situation. 




3A 
4A 
7A 
5A 

C8l 
C82 

C82A 
C83 
C83A 
C84 



ST" 
83 



Infonoatlon on checking the quality of items 
heuristics and algorithms on checking Items 
generaced Items 
the table ot specification 

check It the item set oatcnes the blueprint ' 

itlm^'foroat^^ correctness of items with regard to chosen 

Item set and comments on construction errors 
check the Items on spelling and graooatlcal errors 
Item set and comments from C82 and C83 
check the items on lay-out errors 



rejected/iiaw items and cocnent 
Items ready for use 
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Figure 12. A GIKAPH and TEXTPAGF snowing the checking of items in the 
n^w situation. 
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